System and method for recommending a grammar for a message campaign used by a message optimization system

ABSTRACT

A system and method is provided for recommending a grammar for a message campaign used by a message optimization system. A user specifies parameters for a new campaign, from which a set of statistical design budgets is calculated. The user selects a grammar structure, recommended based on the statistical design budgets, for the campaign. The n-most relevant past campaigns are identified. Semantic tags, associated with each previously used value from the n-most relevant past campaigns and each of a plurality of untested values, are identified and ranked based on past performance. The previously used values are ordered by ranked tag group and then within each tag group, while the untested values are ordered by ranked tag group and then randomly within the tag group. Recommended values are selected from the ranked list of previously used values and untested values depending on the degree of exploration/conservatism indicated by the user.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/474,695 titled “System and Method for Recommending a Grammar for aMessage Campaign Used by a Message Optimization System,” which was filedon May 17, 2012, the contents of which are incorporated by reference asif fully disclosed herein.

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention relates generally to a message optimization system and,more particularly, to a system and method for recommending a grammar fora message campaign used by a message optimization system.

2. Description of the Background Art

Commercial advertising has undergone a significant shift in the pastdecade. Traditional media advertising, taking the form of newspapers,magazines, television commercials, radio advertising, outdooradvertising, and direct mail, etc., has been decreasing as the primarymethod of reaching an audience, especially as related to certain targetdemographics or types of products. New media advertising, in the form ofPopup, Flash, banner, Popunder, advergaming, email advertising, mobileadvertising, etc., has been increasing in prominence.

One characteristic of new media advertising is the need to capture anaudience's (viewers, readers, or listeners) attention with limited text.For example, with a banner or text message, the sponsor of theadvertising message may only have a finite number of characters topersuade its audience to act by clicking on a link, texting back amessage, etc. As a result, companies are increasingly interested in howto optimize their message, and the components in the message, toincrease the message's response rate. International Publication NumberWO 2011/076318 A1 discloses a system and method for optimizing a messageand is incorporated by reference herein in its entirety. In this system,the message is divided into components and multiple values are testedfor each component to determine the best response rates.

The best message optimization system, however, is only as good as itsstarting values. In the past, a user, or campaign manager, wouldmanually choose the starting values. However, an inexperienced campaignmanager may not know what values to start with. Therefore, there is aneed for a system and method for recommending an initial grammar for amessage campaign used by a message optimization system based on the datafrom previous campaigns.

SUMMARY OF THE INVENTION

The present invention is directed to a system and method forrecommending a grammar for a message campaign used by a messageoptimization system. The method includes enabling a campaign manager tospecify one or more parameters for a new campaign. A set of statisticaldesign budgets is calculated based on one or more of the specifiedparameters. In one embodiment, the statistical design budgets arecalculated based on audience size, effect size, and expected responserate.

At least one grammar structure is recommended based on the set ofstatistical design budgets, where the grammar structure specifies aplurality of message component types. A campaign manager is then able toselect one of the recommended grammar structures for the new campaign.The n-most relevant past campaigns are identified based on the specifiedparameters. For each message component type in the selected grammarstructure, a ranked list of previously used values for the componenttype in the n-most relevant past campaigns is generated, where thevalues are ranked at least in part based on performance in the n-mostrelevant past campaigns. For each message component type, a plurality ofvalues is selected to recommend based at least in part on the rankedlist.

The campaign manager is able to reject one or more of the recommendedvalues. In response to the campaign manager rejecting one or more of therecommended values, alternate recommended values are provided for therejected values. In certain embodiments, providing alternate recommendedvalues for the rejected values comprises choosing values from the nextbest performing tag or another candidate value associated with the sametag as the previously rejected value. In one embodiment, the methodfurther comprises evaluating the recommended grammar based on variouscomputed metrics.

In one embodiment, the previously used values are each associated with asemantic tag. Generating the ranked list includes identifying the tagsassociated with the previously used values in the n-most relevantcampaigns. The tags are then ranked based on performance of taggedvalues in the n-most relevant campaigns. The previously used values areordered first by ranked tag group and second, within each tag group, bythe number of time an individual value has been identified as thewinning value in the n-most relevant campaigns.

In one embodiment, the recommend values are selected in part from theranked list of previously used values and in part from a ranked list ofuntested values. A plurality of untested values may be grouped bysemantic tag and ranked according to their semantic tag, where within atag group an untested value is randomly ranked. The percentage ofrecommended untested values may depend on a degree ofexploration/conservatism indicated by the campaign manager.

In one embodiment, enabling a campaign manager to specify parameters fora new campaign includes providing a user interface wherein the campaignmanager is prompted to enter parameters for the campaign. One or moreparameters in the user interface may be associated with a drop down menuthat lists the options available to the campaign manager for theparameter. The parameters may comprise campaign duration, audience sizeand characteristics, expected response rate, effect size, constraints,and objectives of the campaign. The campaign manager may select theobjectives of the campaign via a drop down menu, from a sliding scale,or by inputting a value. In certain embodiments, if the campaign managerdoes not specify certain parameters, default values, empiricallydetermined based on past campaigns, are used.

The present invention provides many unique advantages. For example,multiple campaign managers are able to centrally and efficiently accessthe consolidated, distilled information from each of their individualcampaigns. This allows novice campaign managers to become productivevery quickly and allows individual campaign managers to manage a greaternumber of campaigns at the same time. For example, whereas previously acampaign manager would need at least 2 to 3 hours to run a singlecampaign, a campaign manager using the present system is able to processcampaigns orders of magnitude faster.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1a-1b are a flowchart that illustrates a method for recommending agrammar according to one embodiment of the invention.

FIG. 2 is a flowchart that illustrates a method for recommending agrammar structure according to one embodiment of the invention.

FIG. 3 is a flowchart that illustrates a method for generating a rankedlist for previously tested values according to one embodiment of theinvention.

FIG. 4 is a flowchart that illustrates a method for generating a rankedlist for previously untested values according to one embodiment of theinvention.

FIG. 5 is a block diagram of an exemplary software architecture for arecommendation system according to one embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention provides a system and method for recommending agrammar for a message campaign used by a message optimization system.The message optimization system uses the grammar to automaticallygenerate variations of the message and test at least some of thevariations.

As used herein, a “component” is a functional part of a message.Components may be tangible, such as an introduction, benefit, or a callto action, or they may be intangible, such as the level of formality,the verb tense, or the type of persuasion. Components are filled with“values,” which may be categorized by families. For example, valuesfilling the introduction component may be from the greetings category(e.g., “Congratulations!”, “Hello!”, etc.) or they may be from theurgency category (e.g., “Only Today!”, “Act Now!”, etc.) or from othercategories. The type of components and the values tested for eachcomponent are the “grammar” of the message. The invention as describedherein applies to context-free grammars and grammars that are notcontext-free.

Shown below is a simple example grammar named “sentence”, consisting offour components: “intro”, “benefit”, “product”, and call to action or“cta”. Next to each listed component are values that may fill thecomponent. For example, the introduction component may be filled with“Congratulations!” or “Great news!”.

sentence -> [intro] [benefit] [product] [cta] intro -> Congratulations!| Great news! benefit -> We have a great offer for you! | Take advantageof this unique opportunity! product -> Top up your phone with 5$ and get2$ on us! | For every 5$ top-up we'll give you 2$ extra! cta -> Act nowand top up! | Top-up your phone today and win! | Don't waste time,top-up today!

A “campaign” is the process for testing grammar structures and valuesfor the structures to determine which structures and values receive thebest response rate, where the “response rate” is the acceptance of acall to action (e.g., calling a number, sending an sms to a number,clicking on a weblink, clicking on a mobile weblink, proceeding topurchase, etc.). The “expected response rate” is the rate the campaignmanager expects to get from the target audience for the campaign (e.g.,3%, 5%, etc.). The “effect size” is the percentage difference in theresponse rate that can be created by varying the values of thecomponents. A “campaign manager” is the user of the message optimizationsystem.

FIGS. 1a-1b show the preferred method for recommending a grammar. Thecampaign manager specifies one or more parameters for a campaign (step110). The recommendation system generates a user interface that enablesa campaign manager to enter parameters for a campaign. In oneembodiment, a plurality of configurable parameters is each associatedwith a drop down menu that lists the options available to the campaignmanager for the parameter. Examples of parameters include country,client, language, campaign duration, product, type of offer, promotionchannel, activation/participation channel, whether the campaign is acontinuation or repetition of a previous campaign, audience size andcharacteristics (e.g., gender), expected response rate, effect size,pValues, constraints (e.g., message length constraints, mandatorycomponents, etc.), and grammar quality, etc. In a preferred embodiment,the campaign manager selects the objectives of the campaign (i.e., thedegree of conservatism/exploration or tested versus untested values).The campaign manager may select from a drop down menu, select from asliding scale, or use other means to indicate the campaign objectives.In certain embodiments where the campaign manager does not specifycertain parameters, the system uses default values that are empiricallydetermined based on past campaigns. For example, parameters n or k,discussed below, may be empirically determined and set as system-wideparameters.

The system calculates a set of “statistical design budgets” based onaudience size, effect size, and expected response rate (step 115). Thestatistical design budget for a campaign is the number of components ina message and the number of values to test for each component. Differentvariations can have the same or almost the same statistical designbudget. For example, a grammar with three message components and fourvalues for each component will generate the same number of messagevariations as message with four components and three values to test foreach. The number of values associated with each component can varywithin a grammar. For example, there may be five values for onecomponent and two values for another.

If the effect size is small (i.e., the percentage difference between theresponse rate of two values is small), then the audience size must belarge in order to return statistically significant data. For example, ifComponent Value A has a response rate of 3.0% and Component Value B hasa response rate of 3.3%, the difference in absolute terms is 0.3% or aneffect size of 10%.

A person skilled in the art would understand that an effect size of 0-5%is negligible, such that it would be very hard to differentiate betweenvariations in response rates and inherent noise. An effect size of 5-10%is generally considered very small and would require a very large samplein order to establish the statistical significance. An effect size of10-15% is considered small and would require a relatively large sample,but not as large as the sample required for an effect size of 5-10%. Aneffect size of 15-30% is generally considered “medium” and is the focalpoint of interest in most optimization scenarios. An effect size of 30%or greater is large and usually observed when using extreme componentvalues or with very small response rates. In the above example, thesystem would need a very larger audience size in order to produce usefuldata.

The expected response rate is the response rate (from the targetaudience) that the campaign manager expects to get for the campaign(e.g. 3%, 5%, etc.). In one embodiment, a campaign manager can specifythe effect size and response rate in step 110. If the campaign managerdoes not specify the effect size and the response rate, the system mayuse default values that are empirically determined based on pastcampaigns.

Returning again to FIG. 1, the system recommends one or more grammarstructures (i.e., the type of components and the initial organization ofthe components within the grammar) based on the set of statisticaldesign budgets (step 120). In other words, the system attempts torecommend a grammar structure that fits within the set of statisticaldesign budgets. In the example grammar above, the grammar structure is:sentence->[intro] [benefit] [product] [cta]. In recommending a grammarstructure, the system may consider appropriate components to break themessage up based on established practices in similar campaigns orcommonly recurring components across a plurality of campaigns. If thecampaign manager rejects all of the recommended grammar structures, thesystem recommends additional grammar structures.

If the campaign manager accepts one of the recommended grammarstructures (step 125), then, for each component type in the grammarstructure, the system generates a ranked list of previously used valuesin the n-most relevant past campaigns (step 130). In the preferredembodiment, values are ranked by semantic meaning (i.e., tag applied bythe campaign manager) and, within a semantic grouping, by performance(i.e., response rate). The n-most relevant campaigns are determinedusing the parameters entered by the campaign manager in step 110.

For each component in the selected grammar structure, the system selectsa plurality of values to recommend based at least in part on the rankedlist generated for that component type (step 135). Other factors thatthe system may consider in recommending candidate values are values thatsatisfy any constraints on category or length (i.e., values that do notsatisfy length/category strength are filtered out), winners fromgrammars in similar past campaigns, values that have performed well ingeneral, or values from lists of untested values. The system may alsoconsider synonyms or semantically related values to other wellperforming candidate values from past campaigns. Values across languagesmay be chosen based on their translation. The system may also selectvalues from both a ranked list of previously used values and in partfrom a ranked list of untested values based on the degree ofconservatism/exploration indicated by the campaign manager. The numberof values recommended for each component is based on the parameters ofthe statistical design budget for the selected grammar. The recommendedgrammar is then presented to the campaign manager (step 140).

In one embodiment, the system computes various metrics for therecommended grammar to enable the campaign manager to evaluate therecommended grammar (step 145). Factors affecting the evaluation of therecommended grammar may include the degree of exploration of the grammar(i.e., how many new/untested component values are being tried versus howmany are tried and tested), the “structural spread” of the grammar(i.e., whether the grammar branches out across a wide variety ofsyntactic constructs or whether it narrowly explores a small area), howmuch of the design budget is being spent (i.e., how many messages doesthe experimental design predict we need to send in order to fully testthe grammar), how is the distribution across message components (i.e.,are some components being tested across a wide range of candidate valuesand others only a few), amount of wasted text space (i.e., whether shortcomponents are being compared with long components where the longcomponent values are reserving unnecessary space), how close we are tothe optimal grammar (i.e., sometimes the campaign manager will choose asuboptimal candidate value as the top values from the top performingtags may be overused), degree of repeatability (i.e., how similar is therecommended grammar to the grammars that have recently been used on thisaudience), and degree of novel investment (i.e., whether the system istesting a large number of values in components that generally have fewercombinations or vice versa). The metrics applied to the recommendedgrammars need to be interpreted in light of the parameters of aparticular campaign and the objectives of the campaign manager.

The campaign manager is able to reject recommended component values(step 150). For example, there may be legal or branding issues withcertain expressions that the campaign manager wishes to avoid. If thecampaign manager rejects one or more of the component values (step 155),the system provides alternate values (step 160). For values chosen froma ranked list, the system selects the next best value(s) that satisfiesany specified constraints (e.g., semantic tag constraints, sizeconstraints, etc.). The campaign manager may choose values from the nextbest performing tag or another candidate value associated with the sametag as the previously rejected value. The system then computes thevarious metrics for the recommended grammar with the alternate valuesaccording to step 145. If, however, the campaign manager does not rejectany component values, the system “commits” the grammar for processing(for use in the campaign) (step 165).

In certain embodiments, data from past campaigns is stored in one ormore databases. The data may include the grammar that was used,parameters related to the campaign (e.g., country, client, language,campaign duration, product, type of offer, promotion channel,activation/participation channel, whether the campaign is a continuationor repetition of a previous campaign, audience size and characteristics,expected response rate, effect size, constraints, grammar quality,objectives of the campaign, etc.), the response rate, the correspondingpValue for the response rate, etc. The response rates, pValues, andother data may be automatically entered by a message optimization systemwhen a campaign is run. Typically, a campaign manager manually tagsvalues with semantic tags.

FIG. 2 illustrates a process for recommending a grammar structureaccording to step 120 in FIG. 1. The system first determines whether atleast k of the n-most relevant past campaigns have a grammar within thestatistical design budget (step 210). If not, the system selects agrammar structure from a default template grammar that complies with thestatistical design budget (step 230). If it does, then the systemdetermines whether k is greater than the value 1. If not, the systemselects a grammar structure from the one campaign with a grammar withinthe statistical design budget (step 240). If it is, the system presentsgrammar structures for each of the k relevant campaigns to the campaignmanager and lets the campaign manager select from the presented grammarstructures (step 250).

FIG. 3 illustrates a method for generating a ranked list for previouslytested values according to step 130 in FIG. 1. As a ranked list isgenerated for each component type in the grammar structure selected bythe campaign manager, steps 320-350 of the method are performed for eachcomponent type in the grammar structure. The system identifies then-most relevant campaigns based on parameters entered by the campaignmanager (step 310). From the grammars used in such campaigns, the systemidentifies the various values tested in such campaigns for the relevantcomponent type (step 320). In the preferred embodiment, only campaignsfrom the selected set that include the component type are considered.All of the identified component values are then grouped by the semantictags associated with their values (step 330). The tags in step 330 areranked based on past performance, in terms of response rate, of thevalues within a tag group, from the highest to the lowest, versus othertag groups (step 340). In the preferred embodiment, the past performancedata is limited to data from the n-most relevant campaigns. The tagrankings reflect the performance of the tag group for the relevantcomponent type in the n-most relevant campaigns. A ranked list of allcomponent values (for the applicable component type) from the n-mostrelevant past campaigns is then created, wherein component values areordered first by tag (in order of tag rankings), and then, within eachtag group, by the number of times the individual value has beenidentified as the winning value in the n-most relevant campaigns for therelevant component type (step 350). The values that do not satisfylength constraints or other constraints are filtered out.

In one embodiment, the tags and component values are ranked as follows:

Total Tag 1 Tag 2 Tag 3 . . . Tag t Tag 1 m(1, 1) m(1, 2) m(1, 3) . . .m(1, t) m(1, Total) Tag 2 m(2, 1) m(2, 2) m(2, 3) . . . m(2, t) m(2,Total) Tag 3 m(3, 1) m(3, 2) m(3, 3) . . . m(3, t) m(3, Total) . . . . .. . . . . . . . . . . . . Tag t m(t, 1) m(t, 2) m(t, 3) . . . m(t, t)m(t, Total)

For every cell value m(i, j), where 0<i≤t, 0<j≤t, and t is the number oftags in the system, information about Tag i competing against Tag j iscomputed as follows:

1. Compute the number of times a component text that is tagged with Tagi is chosen over a component text that is tagged with Tag j; compute thenumber of times a component text that is tagged with Tag j is chosenover a component text that is tagged with Tag i; compute the number oftimes the results show that the performance of Tag i and Tag j are notstatistically different.

-   -   2. Using the above values, compute the ratio of wins to losses,        ratio of wins to total, and take the weighted average for the        aggregate score for each cell value.

For every cell value m(i, Total), a summary of the performance of Tag iacross all tags is computed by aggregating all the cell scores in therow for that tag. The number may be zero as the two tags may have nevercompeted against each other.

The above matrix may be computed each time the system recommends a newgrammar because the campaign manager may give different parameters andthe system may select a different set of n-most relevant campaigns.

The system ranks the tags from the highest Total to the lowest Totalscore. For each component type, a list is then created of all thecomponent values grouped by tag and ordered first by tag and then withineach tag by the number of times the individual component value has beenidentified as a winning value in any past grammar in the set of n-mostrelevant campaigns. From this list, the component values are selected tofill the grammar structure based on the design budget (e.g., if thedesign budget calls for four introduction component values, the systemselects the first component value that comes from the top four componenttag groups, filtering out inappropriate values based on constraints).

In one embodiment, if two tag groups are equal in performance (or areuntested relative to each other), the ranking between the two tag groupsis randomly determined. In another embodiment, the ranking is based on apredefined default. In still another embodiment, for breaking tiesbetween tag groups, the system will consider a global ranked list thattakes into account ranking information of all campaigns and not just then-most relevant campaigns.

Depending on the ratio of conservatism to exploration indicated by thecampaign manager, the system may add some untested values into thecampaign. The more conservative the campaign, the more values areselected from the ranked list of tested values from previous, relevantcampaigns, but the more exploratory the campaign, the fewer testedvalues are selected and the more untested, or little-tested values, areselected.

FIG. 4 illustrates a method for generating a ranked list for previouslyuntested values according to step 130 in FIG. 1. As is the case with themethod of FIG. 3, this method is performed for each component type inthe grammar structure selected by the campaign manager. The systemretrieves untested values for the relevant component type from adatabase, where the untested values are each associated with tags in theranked list of tags from step 340 (step 410). All of the untested valuesin step 410 are grouped by the semantic tag associated with each value(step 420). The tags are ranked based on past performance of the taggroup for previously tested values (step 430). In other words, the tagrankings from step 340 in FIG. 3 are used. For the applicable componenttype, a ranked list of all untested component values grouped by tag iscreated (step 440). Component values are ordered first by tag, and then,within each tag group, untested components values are randomly ranked.Values exceeding length or violating other constraints are filtered out.In certain embodiments, an untested value that is not associated with atag in the ranked list of tags from step 340 may be used. These untestedvalues having untested/unranked tags would preferably be groupedtogether separately and used by a campaign manager who desires a greaterdegree of exploration or would like to test certain untested tags. Incertain embodiments, the system may assign a confidence level to theuntested value having a corresponding untested/unranked tag based on thetag's global ranking, which is its ranking across all campaigns and notjust the n-most relevant campaigns.

From the lists generated according to the methods of FIGS. 3 and 4, thesystem recommends values for each of the component types in the grammarstructure. The number of values recommended for each component type isbased on the statistical design budget. In one embodiment, the mix ofpreviously-tested values versus untested values in the recommendationdepends on the level of exploration indicated by the campaign manager.In certain embodiments, the ratio of conservatism to exploration isdetermined by a sliding scale adjusted by the campaign manager in theuser interface. In other embodiments, it is determined by a percentageinputted by the campaign manager. In still other embodiments, the systemuses a default percentage mix (e.g., 25% untested values, 75%previously-tested values).

FIG. 5 illustrates an exemplary system architecture for a RecommendationSystem 500. As a person skilled in the art would understand, the systemarchitecture may be constructed in any number of ways, having more orless modules and different interconnectivity, within the scope of thepresent invention. The methods of FIGS. 1-4 may be implemented in othersystems, and the invention is not limited to system 500.

Client Applications 505 provide a user interface via which users (e.g.,campaign managers) can enter parameters for a new campaign and reviewand/or modify a recommended grammar. The Client Applications 505 may berun on any number of systems and may connect to the RecommendationSystem 500 through any number of channels. For example, the ClientApplications 505 may include web, desktop, or mobile applications.

The Recommendation System 500 has a User Interface (UI) GenerationModule 510, a Recommendation Engine 520, a Ranking Module 530, aRelevant Campaign Retrieval Module 540, a Statistical Design Module 550,a Tagging Module 560, a Grammar Evaluation Module 570, and a DatabaseInterface 580.

The UI Generation Module 510 provides a user interface between theclient applications 505 and the Recommendation Engine 520. TheRecommendation Engine 520 recommends an initial grammar for the messagecampaign used by the message optimization system. The Ranking Module 530ranks a plurality of semantic tags based on the past performance of thepreviously used values associated with each semantic tag. The RankingModule 530 then orders the previously used values first by ranked taggroup and then within each tag group. The plurality of untested valuesis grouped by semantic tag, but then randomly ordered within the taggroup.

The Relevant Campaign Retrieval Module 540 retrieves the relevant pastcampaigns given the set of user specified input parameters. TheStatistical Design Module 550 produces a set of statistical designbudget options. The Tagging Module 560 interacts with the UI GenerationModule 510 to allow the campaign manager to tag the components withsemantic tags. The Grammar Evaluation Module 570 computes metrics for arecommended grammar given the set of input parameters. The DatabaseInterface 580 interfaces with one or more databases 590, which functionsto store past campaign data (e.g., the grammar for each campaign, theresponse rate for each tested value, the pValue indicating thestatistical significance of the difference between tested values for acomponent, etc.).

The methods described with respect to FIGS. 1-4 are embodied in softwareand performed by a computer system executing the software. A personskilled in the art would understand that a computer system has a memoryor other physical, computer-readable storage medium for storing softwareinstructions and one or more processors for executing the softwareinstructions.

As will be understood by those familiar with the art, the invention maybe embodied in other specific forms without departing from the spirit oressential characteristics thereof. Accordingly, the above disclosure ofthe present invention is intended to be illustrative and not limiting ofthe invention.

1. A method performed by a computer system for recommending a grammarfor a message campaign used by a message optimization system, the methodcomprising: providing a user interface that enables a campaign managerto specify one or more parameters for a new campaign; recommending atleast one grammar structure from one or more past campaigns that arebased on the campaign parameters or from a default grammar in the eventthat none of the past campaigns has a grammar that complies with thecampaign parameters, the grammar structure specifying a plurality ofmessage component types; providing a user interface that enables acampaign manager to select one of the recommended grammar structures forthe new campaign; for each message component type in the selectedgrammar structure, generating a ranked list of previously-used valuesfor the component type in the one or more past campaigns, wherein thepreviously-used values are each associated with a semantic tag andgenerating the ranked list comprises: identifying the semantic tagsassociated with the previously-used values in the one or more pastcampaigns, wherein each semantic tag identifies the semantic meaning ofthe associated value, creating a list of the previously-used values inthe one or more past campaigns grouped by semantic tag, ranking groupsof semantic tags based on performance in the one or more past campaignsof the previously-used values within a tag group versus other taggroups, and ordering the previously-used values first by their rankedtag group and second, within each tag group, by the number of times anindividual value has been identified as the winning value in the one ormore past campaigns; for each message component type, selecting aplurality of values to recommend testing based at least in part on theranked list of previously-used values; enabling the campaign manager toreject one or more of the recommended values; in response to thecampaign manager rejecting one or more of the recommended values,providing alternate recommended values for the rejected values; andgenerating variations of a message to test based on the grammarstructure and values accepted by the campaign manager.
 2. The method ofclaim 1, wherein the recommended values are selected in part from theranked list of previously-used values and in part from a ranked list ofuntested values.
 3. The method of claim 2, wherein a plurality ofuntested values is grouped by semantic tag and ranked according to theirsemantic tag, wherein within a tag group an untested value is randomlyranked.
 4. The method of claim 2, wherein the percentage of recommendeduntested values depends on a degree of exploration/conservatismindicated by the campaign manager.
 5. The method of claim 1, wherein therecommended values are also selected in part from a group of untestedvalues having an associated untested/unranked semantic tag.
 6. Themethod of claim 5, wherein each untested value in the group of untestedvalues is associated with a confidence level based on the global rankingof the untested value across all campaigns.
 7. The method of claim 1,wherein one or more parameters in the user interface is associated witha drop-down menu that lists the options available to the campaignmanager for the parameter.
 8. The method of claim 1, wherein theparameters comprise campaign duration, audience size andcharacteristics, expected response rate, effect size, constraints, andobjectives of the campaign.
 9. The method of claim 8, wherein thecampaign manager selects the objectives of the campaign via a drop-downmenu, from a sliding scale, or by inputting a value.
 10. The method ofclaim 1, wherein if the campaign manager does not specify certainparameters, default values, empirically determined based on pastcampaigns, are used.
 11. The method of claim 1, further comprisingevaluating the recommended grammar based on various computed metrics.12. The method of claim 1, wherein providing alternate recommendedvalues for the rejected values comprises choosing values from the nextbest performing tag group or another candidate value associated with thesame semantic tag as the previously rejected value.
 13. A non-transitorycomputer-readable medium comprising code that, when executed by acomputer system, enables the computer system to perform the followingmethod for recommending a grammar for a message campaign used by amessage optimization system, the method comprising: enabling a campaignmanager to specify one or more parameters for a new campaign;recommending at least one grammar structure from one or more pastcampaigns that are based on the campaign parameters or from a defaultgrammar in the event that none of the past campaigns has a grammar thatcomplies with the campaign parameters, the grammar structure specifyinga plurality of message component types; enabling a campaign manager toselect one of the recommended grammar structures for the new campaign;for each message component type in the selected grammar structure,generating a ranked list of previously-used values for the componenttype in the one or more past campaigns, wherein the previously-usedvalues are each associated with a semantic tag and generating the rankedlist comprises: identifying the semantic tags associated with thepreviously-used values in the one or more past campaigns, wherein eachsemantic tag identifies the semantic meaning of the associated value,creating a list of the previously-used values in the one or more pastcampaigns grouped by semantic tag, ranking groups of semantic tags basedon performance in the one or more past campaigns of the previously-usedvalues within a tag group versus other tag groups, and ordering thepreviously-used values first by their ranked tag group and second,within each tag group, by the number of times an individual value hasbeen identified as the winning value in the one or more past campaigns;for each message component type, selecting a plurality of values torecommend testing based at least in part on the ranked list ofpreviously-used values; enabling the campaign manager to reject one ormore of the recommended values; in response to the campaign managerrejecting one or more of the recommended values, providing alternaterecommended values for the rejected values; and generating variations ofa message to test based on the grammar structure and values accepted bythe campaign manager.
 14. The non-transitory computer-readable medium ofclaim 13, wherein the recommended values are selected in part from theranked list of previously-used values and in part from a ranked list ofuntested values.
 15. The non-transitory computer-readable medium ofclaim 14, wherein a plurality of untested values is grouped by semantictag and ranked according to their semantic tag, wherein within a taggroup an untested value is randomly ranked.
 16. The non-transitorycomputer-readable medium of claim 14, wherein the percentage ofrecommended untested values depends on a degree ofexploration/conservatism indicated by the campaign manager.
 17. Thenon-transitory computer-readable medium of claim 13, wherein therecommended values are also selected in part from a group of untestedvalues having an associated untested/unranked semantic tag.
 18. Thenon-transitory computer-readable medium of claim 17, wherein eachuntested value in the group of untested values is associated with aconfidence level based on the global ranking of the untested valueacross all campaigns.
 19. The non-transitory computer-readable medium ofclaim 13, wherein enabling a campaign manager to specify parameters fora new campaign comprises providing a user interface wherein the campaignmanager is prompted to enter parameters for the campaign.
 20. Thenon-transitory computer-readable medium of claim 19, wherein one or moreparameters in the user interface is associated with a drop-down menuthat lists the options available to the campaign manager for theparameter.
 21. The non-transitory computer-readable medium of claim 13,wherein the parameters comprise campaign duration, audience size andcharacteristics, expected response rate, effect size, constraints, andobjectives of the campaign.
 22. The non-transitory computer-readablemedium of claim 21, wherein the campaign manager selects the objectivesof the campaign via a drop-down menu, from a sliding scale, or byinputting a value.
 23. The non-transitory computer-readable medium ofclaim 13, wherein if the campaign manager does not specify certainparameters, default values, empirically determined based on pastcampaigns, are used.
 24. The non-transitory computer-readable medium ofclaim 13, further comprising evaluating the recommended grammar based onvarious computed metrics.
 25. The non-transitory computer-readablemedium of claim 13, wherein providing alternate recommended values forthe rejected values comprises choosing values from the next bestperforming tag group or another candidate value associated with the samesemantic tag as the previously rejected value.
 26. A computer system forrecommending a grammar for a message campaign used by a messageoptimization system, the system comprising: a processor; a memorycoupled to the processor, wherein the memory stores instructions that,when executed by the processor, causes the system to perform theoperations of: enabling a campaign manager to specify one or moreparameters for a new campaign; recommending at least one grammarstructure from one or more past campaigns that are based on the campaignparameters or from a default grammar in the event that none of the pastcampaigns has a grammar that complies with the campaign parameters, thegrammar structure specifying a plurality of message component types;enabling a campaign manager to select one of the recommended grammarstructures for the new campaign; for each message component type in theselected grammar structure, generating a ranked list of previously-usedvalues for the component type in the one or more past campaigns, whereinthe previously-used values are each associated with a semantic tag andgenerating the ranked list comprises: identifying the semantic tagsassociated with the previously-used values in the one or more pastcampaigns, wherein each semantic tag identifies the semantic meaning ofthe associated value, creating a list of the previously-used values inthe one or more past campaigns grouped by semantic tag, ranking groupsof semantic tags based on performance in the one or more past campaignsof previously-used values within a tag group versus other tag groups,and ordering the previously-used values first by their ranked tag groupand second, within each tag group, by the number of times an individualvalue has been identified as the winning value in the one or more pastcampaigns; for each message component type, selecting a plurality ofvalues to recommend testing based at least in part on the ranked list ofpreviously-used values; enabling the campaign manager to reject one ormore of the recommended values; in response to the campaign managerrejecting one or more of the recommended values, providing alternaterecommended values for the rejected values; and generating variations ofa message to test based on the grammar structure and values accepted bythe campaign manager.